In enterprise environments with large data processing requirements, reducing the total cost of ownership is a significant consideration. For example, to optimize hardware utilization and to reduce the number of servers required, it may be desirable to consolidate multiple application processes on a single server, for example by utilizing virtualization or other techniques to host multiple services on a single server. In another example, it may be desirable to utilize a clustered file system to provide shared consolidated storage for several servers.
Such consolidated server environments will often experience heavy read and write loads with many concurrent data requests. To service such data requests in a timely fashion and to meet application performance requirements, it may be preferable to use expensive high-speed media such as solid state disks. Accordingly, maximizing utilization of available data storage becomes a much larger factor in lowering the total cost of ownership.
One approach to maximize data storage utilization is to compress data, which can provide significant space savings at the cost of increased processor overhead. Another approach is to provide deduplication, where redundant copies of data are eliminated and replaced with references to a single copy of the data. Both approaches may also be combined and may be especially effective for consolidated server environments.
To provide higher performance and to optimize free space management, features such as compression and deduplication are typically tightly integrated into file systems at a low level. However, many existing file systems do not provide native integrated support for compression and deduplication. Moreover, when a system is already using a particular file system that lacks native support for compression and deduplication features, it is often not feasible or practical to migrate to a different file system having such feature support, particularly for production systems restricted to specific well-known working environments. While open source file systems may allow for the possibility of adding new features, such an undertaking may require significant development and testing resources to ensure proper integration, compatibility, and stability.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.